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Amendment to the Claims : 

This listing of claims replaces all prior versions, and 
listings, of claims in the application: 

1, (cancelled) 

2. (currently amended) [[A]] The m ethod of claim 12 
Gomprioing ; 

generating a three dimensional model of an environment from 
range oenoor information rcprcocnting a height field for the 
environment; 

tracking orientation information of at least one image 
oenoor in the environment with rcapcGt to the three dimensional 
model in real time; 

projecting real time video imagery information from the at 
Icaot one image oenoor onto the three dimenoional model baoed on 
the tracked orientation information; and 

vioualiaing the three dimenoional model with the projected 
real time video imagery; 

wherein generating the three dimensional model comprises 
identifying a structure in the range sensor information, 
identifying different sections of the structure, selecting 
geometric primitives for the different sections of the structure 
based at least in part on input from a person regarding 
different shapes of the different sections, and parametric 



2 



Serial No.: 10/676,377 Attorney's Docket No.: 06666-156001/USC-3345 

fitting of the geometric primitives to the range sensor 
information. 

3. (previously presented) The method of claim 2, wherein 
the selecting geometric primitives comprises selecting 

geometric primitives from a group including a sphere primitive 
and a cuboid primitive; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information. 

4. (original) The method of claim 2, wherein the at least 
one image sensor comprises multiple image sensors, and 
generating the three dimensional model further comprises 
refining the three dimensional model based on object surfaces 
mapped from images acquired by the image sensors . 

5. (currently amended) The method of claim 12^ [[1]]/ 
wherein generating the three dimensional model further comprises 
projecting and resampling points in the range sensor information 
onto a regular grid at a user-defined resolution to produce the 
height field. 

6. (original) The method of claim 5, wherein generating 
the three dimensional model further comprises processing the 
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height field using hole-filling and tessellation to generate a 
triangle mesh representation of the three dimensional model. 

7. (original) The method of claim 2, wherein the range 
sensor information comprises light detection and ranging (LIDAR) 
sensor data from an active airborne laser sensor. 

8. (currently amended) The method of claim 12^ [[1]]/ 
wherein tracking orientation information comprises tracking 
position and orientation information of the at least one image 
sensor by estimating a camera pose based at least in part on 
three dimensional parameters of point and line structures 
visible in the camera image, and projecting real-time video 
imagery information comprises projecting real-time video imagery 
information from the at least one image sensor onto the three 
dimensional model based on the tracked orientation information. 

9- (original) The method of claim 8, wherein tracking 
.position and orientation information of the at least one image 
sensor further comprises processing data from a tracking sensor 
system that integrates visual input, global navigational 
satellite system receiver input, and inertial orientation sensor 
input . 

10. (currently amended) The method of claim 12^ [[1]], 
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wherein the at least one image sensor comprises multiple image 
sensors, and projecting the real-time video imagery information 
comprises projecting multiple video streams from the multiple 
image sensors onto the three dimensional model . 

11. (cancelled) 

12. (previously presented) A method comprising: 
generating a three dimensional model of an environment from 

range sensor information representing a height field for the 
environment ; 

tracking orientation information of at least one image 
sensor in the environment with respect to the three dimensional 
model in real-time; 

projecting real-time video imagery information from the at 
least one image sensor onto the three dimensional model based on 
the tracked orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery; 

wherein projecting the real-time video imagery information 
comprises generating a depth map image from a video sensor 
viewpoint, and projective texture mapping the real-time video 
imagery information onto the three dimensional model conditioned 
upon visibility as determined from the generated depth map 
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image ; and 

wherein generating the depth map image and projective 
texture mapping the real-time video imagery information are 
performed using a one -pass approach on graphics hardware that 
supports SGI OpenGL extensions. 

13. (currently amended) The method of claim 12 [[1]], 
wherein visualizing the three dimensional model comprises: 

video-projecting onto a display screen using a stereo 
video -pro j ec t or ; and 

coupling a rendering viewpoint to a user's head position 
using data from a tracker. 

14. (cancelled) 

15. (currently amended) The system of claim 22^ [[14]], 
wherein the at least one image sensor comprises multiple image 
sensors . 

16. (original) The system of claim 16, wherein the real- 
time video imagery information comprises pre-recorded real-time 
video imagery information. 

17. (original) The system of claim 15, wherein the dynamic 
fusion imagery projection component bases the real-time video 
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imagery projection on a viewpoint separate from viewpoints 
associated with the multiple image sensors. 

18. (currently amended) An augmented virtual environment 
The system of claim 22 comprioing : 

a model conotruction component that gcncratoo a three 
dimcnoional model of an environment from range ocnaor 
information rcprcocnting a height field for the environment; 

a dynamic fuoion imagery projection component that project a 
real time video imagery information from at Icaat one image 
oenoor onto the three dimcnoional model baoed on orientation 
information of the at leaot one image ocnoor tracked in the 
environment with reopect to the three dimensional model in real 
time ; — aftd 

a viaualigation sub oystem that viaualizco the three 
dimcnoional model with the projected real time video imagery; 

wherein the model construction component performs 
operations comprising: 

identifying a structure in the range sensor information; 

identifying different sections of the structure; 

selecting geometric primitives for the different sections 
of the structure based at least in part on input from a person 
regarding different shapes of the different sections; and 

parametric fitting of the selected geometric primitives to 
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the range sensor information. 

19. (original) The system of claim 18, wherein the 
operations further comprise: 

projecting and resampling points in the range sensor 
information onto a regular grid at a user-defined resolution to 
produce the height field; and 

processing the height field using hole-filling and 
tessellation to generate a triangle mesh representation of the 
three dimensional model . 

20. (currently amended) The system of claim 22_ [[14]], 
wherein the at least one image sensor comprises multiple image 
sensors, and the system further comprises a model refinement 
component that refines the three dimensional model based on 
object surfaces mapped from images acquired by the image 
sensors . 

21. (currently amended) The system of claim 22^ [[14]], 
further comprising a traclcing sensor system that integrates 
visual input, global navigational satellite system receiver 
input, and inertial orientation sensor input to obtain position 
and orientation information of the at least one image sensor, 
and the dynamic fusion imagery projection component projects the 
real-time video imagery information based on the position and 
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orientation information. 

22. (previously presented) An augmented virtual 
environment system comprising: 

a model construction component that generates a three 
dimensional model of an environment from range sensor 
information representing a height field for the environment; 

a dynamic fusion imagery projection component that projects 
real-time video imagery information from at least one image 
sensor onto the three dimensional model based on orientation 
information of the at least one image sensor tracked in the 
environment with respect to the three dimensional model in real- 
time; and 

a visualization sub-system that visualizes the three 
dimensional model with the projected real-time video imagery; 

wherein the visualization sub-system comprises the dynamic 
fusion imagery projection component and graphics hardware that 
supports SGI OpenGL extensions, and uses a one-pass approach on 
the graphics hardware to generate a depth map image from a video 
sensor viewpoint and projective texture map the real-time video 
imagery information to project the real-time video imagery 
information conditioned upon visibility as determined from the 
generated depth map image. 
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23. (currently amended) The system of claim 22^ [[14]], 
wherein the visualization sub-system comprises: 

a stereo video-projector; and 
a tracker. 

24. (cancelled) 

25. (previously presented) The method of claim 29, wherein 
the surface comprises a two dimensional surface. 

26. (original) The method of claim 25, wherein placing the 
two dimensional surface comprises: 

casting a ray from an optical center, corresponding to the 
real-time video imagery information, to a bottom point of the 
moving region in an image plane in the three dimensional model; 
and 

determining a position, an orientation and a size of the 
two dimensional surface based on the ray, a ground plane in the 
three dimensional model, and the moving region. 

27. (cancelled) 

28. (cancelled) 

29. (currently amended) A method comprising: 
obtaining a three dimensional model of an environment; 
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identifying in real time a region in motion with respect to 
a background image in real-time video imagery information from 
at least one image sensor having associated position and 
orientation information with respect to the three dimensional 
model/ the background image comprising a single distribution 
background dynamically modeled from a time average of the real- 
time video imagery informations- 
placing a surface that corresponds to the moving region in 
the three dimensional models- 
projecting the real-time video imagery information onto the 
three dimensional model, including the surface, based on the 
position and orientation information; and 

visualizing the three dimensional model with the projected 
real-time video imagery; 

wherein identifying a region in motion in real time 
comprises subtracting the background image from the real-time 
video imagery information, identifying a foreground object in 
the subtracted real-time video imagery information, validating 
the foreground object by correlation matching between identified 
objects in neighboring image frames, and outputting the 
validated foreground object; 

wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time 
video imagery information using a histogram-based threshold and 
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a noise filter; 

wherein identifying a region in motion in real time further 
comprises estimating the background image by modeling the 
background image as a temporal pixel average of at Icaot five 
recent image frames in the real-time video imagery information. 

30. (previously presented) The method of claim 29, further 
comprising tracking the position and orientation information of 
the at least one image sensor in the environment with respect to 
the three dimensional model in real-time. 

31. (original) The method of claim 30, wherein obtaining a 
three dimensional model of an environment comprises generating 
the three dimensional model of the environment from range sensor 
information representing a height field for the environment. 

32. (cancelled) 

33. (previously presented) The system of claim 37, wherein 
the surface comprises a two dimensional surface. 

34. (original) The system of claim 33, wherein the object 
detection and tracking component places the two dimensional 
surface by performing operations comprising: 

casting a ray from an optical center, corresponding to the 
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real-time video imagery information, to a bottom point of the 
moving region in an image plane in the three dimensional model; 
and 

determining a position, an orientation and a size of the 
two dimensional surface based on the ray, a ground plane in the 
three dimensional model, and the moving region. 

35. (cancelled) 

36. (cancelled) 

37. (currently amended) An augmented virtual environment 
system comprising: 

an object detection and tracking component that identifies 
in real time a region in motion with respect to a background 
image in real-time video imagery information from at least one 
image sensor having associated position and orientation 
information with respect to a three dimensional model, the 
background image comprising a single distribution background 
dynamically modeled from a time average of the real-time video 
imagery information, and places a surface that corresponds to 
the moving region with respect to the three dimensional model; 

a dynamic fusion imagery projection component that projects 
the real-time video imagery information onto the three 
dimensional model, including the surface, based on the position 
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and orientation information; and 

a visualization sub-system that visualizes the three 
dimensional model with the projected real-time video imagery; 

wherein the object detection and tracking component 
identifies the moving region by performing operations comprising 
subtracting the background image from the real-time video 
imagery information, identifying a foreground object in the 
subtracted real-time video imagery information, validating the 
foreground object by correlation matching between identified 
objects in neighboring image frames, and outputting the 
validated foreground object; 

wherein identifying a foreground object comprises 
identifying the foreground object in the subtracted real-time 
video imagery information using a histogram-based threshold and 
a noise filter; and 

wherein identifying a region in motion in real time further 
comprises estimating the background image by modeling the 
background image as a temporal pixel average of at Icaot five 
recent image frames in the real-time video imagery information, 

38, (previously presented) The system of claim 37, further 
comprising a tracking sensor system that integrates visual 
input, global navigational satellite system receiver input, and 
inertial orientation sensor input to obtain the position and the 
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orientation information associated with the at least one image 
sensor in real time in conjunction with the real-time video 
imagery. 

39. (original) The system of claim 38, further comprising 
a model construction component that generates the three 
dimensional model of an environment from range sensor 
information representing a height field for the environment. 

40. (cancelled) 

41. (cancelled) 

42. (cancelled) 

43. (cancelled) 
44 - (cancelled) 

45. (previously presented) The method of claim 3, wherein 
the selecting geometric primitives comprises selecting 
geometric primitives from the group including the sphere 
primitive, the cuboid primitive, a hollow-cuboid primitive, and 
a roof primitive, the roof primitive comprising connected 
symmetric slope primitives; and 

the parametric fitting comprises performing constrained 
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best fitting of the 'geometric primitives to the range sensor 
information without ground plan information for the structure. 

46. (previously presented) The system of claim 18, wherein 
the selecting geometric primitives comprises selecting 

geometric primitives from a group including a sphere primitive 
and a cuboid primitive; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information. 

47. (previously presented) The system of claim 46, wherein 
the selecting geometric primitives comprises selecting 

geometric primitives from the group including the sphere 
primitive, the cuboid primitive, a hollow-cuboid primitive, and 
a roof primitive, the roof primitive comprising connected 
symmetric slope primitives; and 

the parametric fitting comprises performing constrained 
best fitting of the geometric primitives to the range sensor 
information without ground plan information for the structure. 

48. (cancelled) 

49. (cancelled) 
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